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I. SOME TASKS AND COMPETENCIES 

FREQUENTLY REQUIRED IN EDUCATIONAL RESEARCH 
AND EVALUATION 

Many educational evaluatora and researehers are handioaj^ed by lack of 
adequate skill and knowledge about substantive and methodological requisitee in 
their areas of investigation. Even the researehers and evaluators who are 
initially well trained for their reepeotive roles are soon faced with the possibility 
of obsolescence because of the scarcity of relevant inservice training programs 
In their area of specialisation* Without adequate opportunities for continued 
educationj such persons often find it difflault to keep abreast of new substantive 
and methodological developments in their fields* 

This problem was recognized early by the American Educational. Research 

Association (AERA) which has worked for almost a decade to otU v inservice 

training opportunities to its members and other professionals engaged In 

research and research-Talated activities in education* ^ As part of this training 

effortj AERA empaneled a Task Force on Research Training^ and charged it 

ipopham (1974) provided an excellent summary of AERA*s efforts in 
training during the past decade. 

2The Task Force, chaired in 1969 by Robert Gagne and in 1970-71 
by the author, included the following membersi Abbott L. Ferriss, Myles J. 
Frledmiui, William J, Gepharti John E. Hopkins^ Reginald L* Jones, Jason 
MiUm^p Harold E, Mltzel, Ellis B, Page, W. James Popham, Ernst 2. 
Rothkopf, and Sam D* Sleber, ' 
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mth (1) building a relev^t knowledge base about training needs and problenaa 
and (2) developing procedures to attack these problems* The Task Force 
quickly became aware that one of the most serious impediments to planning 
and conducting training programs for research and [evaluation personnel in 
education is lack of knowledge about which particular competencies are most 
important in these inquiry activities* Knowledge about such competenoies was 
needed before AERA could determine whether its training programs were effeo- 
tive In ■correcting the deficiencies noted above, 

/ 

During its two years of study, the Task Force carried out a series of 
three USOE ^supported activities designed to identify specific skills and knowledg 
required in educational research and evaluation. ^ The first activity (Worthen & 
Gagne, 1969) included two steps* First , the Taik Force drafted lists of skills 
they thought were necessary in conducting research and evaluation In education* 
^cond, the lists of skills were "reality tested" through interviews with 
60 employers and supervisors of research and evaluation persomiel to determin© 
which skills they viewed as important in condUGtlng research and evaluation in 
the interviewee^s agency, which they saw as unlmportanti and whether skills 
they perceiyad as important had been omitted from the list. 

The second activity was not strictly empirical in nature, although It 
used as input the results yielded by the interviews described above and depended 
on informal observation for the rest. The earlier lists were reviewed and found 
to eKcIuide some competencies the reviewers (Glass & Worthen, 1970) noted 

^Although not reported here, the Task Force also made ml effort to 
Identify competencies in educational development and diffusion, which are 
reported in Worthen & Gagnf (1969) and Anderson, Bt^, (1B11)» 
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to be frequefttly required in their own educational inquiry as well as In itudleB 
conducted by colleagues with whose work they were wf 11 adquainted. The 
addition of these supplemental competencieg resulted in a revised list of ikills 
and knowledge viewed by iti authors as important for educational researchere 
and ©valuators* 

m the third activity (Anderson, et , 1971), a task analysii of reiearch 
and evaluation activities in 13 agencies wae conducted. These agencies included 
umversitleB, research and development denters* private research agencies, 
regional educational laboratories, mid public BChools rated by the Task Force 
members as doing high quality work, ^ One hundred and nine research and 
evaluation workers In those agencies were studied to determine what tasks they 
were called upon to perform in their work and what competencies (skills and 
knowledge) v^em required to perform them. Factor analyses of the results 
yielded several clusters of common research and evaluation tasks and 
competencies. 

Methods used in each of the Task Force activities are described In 
detail in the 1971 study referenced above and will not be repeated here, 

A Syrnthesls of the Task Force Analyses 

The purpose of this paper is to synthesize the results of the three 
previous Task Force efforts Into a single list of comiretenoies frequently 

^These ratings depended on raters experienced with the agencies through 
profeeslonal activities such as consultation with agency personnel, review of 
products, and the like* 
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needed In education^ research and evaluaUon. " Each of the thrae earlier 
activitiee depended on a different approach to Identifying such competencies 
and each approach carried with it different assets and deficits. To reduce 
unique error contributed by any one source, the tasks and conapetenoies 
entered in the synthesis are those which are identifiable in at least two of 
the earlier ^alyses (and in a majority of instances all three). Given this 
effort to draw ouc the most common threads, this synthesis seems likely to 
provide a more accurate depiction of competenoies commonly used in the 
field thaii is true of any of the previous individual efforts on which -t depends. 

The eyntliesis has resulted In a list of 2S general tasks which researchers 
and/or e valuators xraquently must perform* For most of these tasks, competenoies 
necessary to perform the task were also identified, ^^Competencies*^ is used 
in this paper to refer both to specific skills Md^owledge, 

Four caveats are necessary, Flrsft there is no olaim that the 25 ^neral 
tasks and attendaiit eompetencies Include all the important tasks or competenoies 
neceasa^ in the broad range of methodolo^ which might be employed In 
educational research or evaluation. For examples few skills of the ethnographer 
turn up in these lists* Such exclusions should not be construed as suggesting 
that such skills are unimportant, only that they were not found to be frequently 
required in the work of the broad sample of researchers and ©valuators in the 

5lt might be noted that not all InveBtlgatlon of research and evaluation 
competencies has been conducted under the aegis of AERA sponsorship, Owens 
(1968), CoUer (1970), and Schalock & Sell (1972) completed similar analyses. 
In gener^, the resultB of those studies do not differ significantly from those 
conducted by the Task Force, and no attempt has been made to include results 
of these studies in the present synthesis. 
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analysei from which the synthesis is drawn* Further, there are obviously 
persooal characteristics (e* g^. # attitudes, value atmctures, peraon^lQr traits) 
which are important to the successM conduct of research w^d ev^uatlon work but 
which were not included in ^y of the sources on which the present synthesis 
depends* 

Second, there are no competencies Hated for some taeks* whereas other 
tasks subsume long lists of relevant competencies. This unevenness results from 
the fact that praotictog researchers Md evaluators studied In the 1969 and 1971 
analyses md the authors who completed the 1970 analysis all seemed better 
able to articulate the competencies required to perform some tasks than they 
were for others* It may be that resah^rchers and evaluators perform these latter 
tasks intuitively and have not anriyzed them suffieiently to be clear about their 
specific elements* In the case of such "under-understood" tasks, it should not 
be surprising that few relevant skills and knowledge come easily to the mind. 
Rather than hazarding guesses to fill these gaps, it seems advisable to await 
further research efforts on essential competencies in the hope that these 
areas will be further illuminated. 

Third, no effort has been made to specifically label each task and 
competency as belonging more to research than to evaluation or vice versa* 
It has been argued elsewhere (Glass & Worthen, 1970b) that research and 
evaluation differ in important ways (e, g, , in the purpose for which they are 
conducted), but methodologically they are often similar, depending to a con- 
siderable degree on the same type of knowledge and skllL Most of the tasks 
and competencies in the following lists are relevant to both research and 
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evaluation. In oases wherB tasks are relevant to one but not the other, this 
distinction is made plain in the discuislon sections. 

Fourth, there is no implicaticn that all tasks are involved in Bvety 
inquiiy or that all of the tasks must necessarily be carried out by any one 
individual. As in most human endeavors, teamwork is a boon to much 
research and evaluation. Only the lone wolf researcher or evaluator who chooses 
(or is forced) to work without benefit of colleagueship or expert consultation would 
ever need to possess a high degree of competence in all of the relevant areas 
listed below* and then only if it were necessaiy to conduct all the types of studies 
which would bring these particular tasks Mid competencies into play. However, 
it is also true that the more important competencies an Individual possesees, the 

less asslstarce must be sought from others—a happy state Indeed, An Indlvldu^ 
proficient in more of the important competencies is more flexible in working in 
a variety of research and evaluation settings than an individual proficient in a 
smaller subset of the important aompetancles wid tlherefore limited to performing 
in settings where that narrow expertise Js relevant. However, with the increasing 
sophistication and specialization in the various fields on which educational evaluation 
and research depend, it seems ineraasingly unlikely that any single individual ' 
will be such a paragon as to be highly skilled in ^1 of the importMt areas. The 
more reasonable stance seems to be to assert only that the full range of relevant 
competencies be either possessed by or readily available to whatever entity 
is carding out the work—whether that entity be an individual, a team, or an 
organization. 

10 



n. TWENTY-FIVE GENERAL RESEARCH AND EVALUATION 
TASKS AND RELATED COMPETENCIES^ 



Eaoh of the 25 tasks on this and siibEequent pages Is followed by (a) a 
brief elaboration of why that task is important to the researeher and/or 
eyaluator and (b) a listing of any competencies whiah seem on thi basis of the 
previous analyseB to be necessaiy to performance of the task, 

1* Obtaining Information about an area to be researohed or a phenomenoii 
to be evaluated. 

Knowledge of an, area of study Is a neaeBsai^ preaursor to research 
activities, just as evaluation Judgments are neeessarlly preceded by an 
understanding of precisely what phenomenon la being judged. Obtaining 
information from antecedents In science and practioe depends on a variety 
of specific competencies. 

1, 1 Knov;ledge of how formal search procedures can be used to obtain 
Information, 

1. 2 Ability to use library research techniques. 

1,3 Ability to use ERIC Bsxd other Information retrieval aystems, 

1, 4 Knowledge of how to obtain information through Informal means, 

such as Identifying and contacting others working in the same area, 

^Portions of the narrative in this section draw on an earlier paper by 
Glass & Worthen (1970). 
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Drawing implications from results of prior researoh and praGtlce , 
No research endeavor stands alone * and no researcher can afford to Ignore 
or slight the activity of carefully studying the literature of his or her field 
and drawing the appropriate inferences for the future course of reiearoh in 
that diicipline. Whether a historian, a philoBopher, a psychologisti a 
sociologist, or a statistician, the researcher must be able to interpret, 
evaluate and synthesize the relevant literature* The evaluator also draws 
on prior research, especially as a basis for determining standards against 
which a judgment will be made* Study of various ways in which a partleular 
education treatment has been applied in the past is important to insure that 
it is not unwittingly evaluated in its weakest case or against irrelevant 
or unreasonable standards* 

2, 1 Ability to review and evaluate research and research-related reports* 

2, 2 Ability to review and evaluate relevant educational practices* 

2.3 Ability to draw correct inferences, conclusionB, or generalizations* 

2* 4 Ability to synthesize or summarize eKtant knowledge. 

Conce ptualizing the research problem or defininR the object of the evaluation . 

This task is probably both the most Important and the most neglected aspect 

of most research and evaluation studies. The researcher must be able 

to identify significant problems posed by the tradition of Inquiry and the 

accumulated works of both predecesBors and contemporaries* The 

evaluator^s problem Is gc rally set by the client, but It must nonetheless 

be defined as sharply as that of the researcher. 

3.1 Ability to Identify and articulate the problem in a research study* 

3.2 Ability to define precisely the phenomenon to be judged in an 
evaluation study. ^ 
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4* Selecting aja appropriate Inquiry etratea^ for addressing the reBearch or 
evaluation problem^ 

Far too often researeh or evaluation Is based on a particular Ingulf ' 

approach with which the researcher or evaluator is most familiar, even 

when that approach is obviously ineffiulent or Inappropriate for ttie 

particular researeh or evaluation problem. Methodological breadth is 

importiuat to research and critical to evaluation^ where the ellfint sets 

the problems, thereby eliminating the luxuiy the researcher sometimes 

enjoys of defining problems in an area where he or she is ttie most 

conafortable* In evaluation, it is Important to make thoughtftil and 

informed choices among the gener^ approaches afforded by the various 

disciplines* ^ ' ' - 

4* 1 Knowledge of the variety of common inquiry strategies in education 

(e.g., depiction studies I correlational studies, experimental studies). 

4. 2 Knowledge of Inquiry strategies In other disciplines (e. g* , philosophical 
analysis, historiography). 

5* Formulating hypotheses or questions to be answered by the study . 

This activity is the hallmark of rigorous inquiry, whether^ empirical or 
not* Hence, formulating hypotheses or research questions is important- 
for all types of educational research* Tr tevaluationi the evaluative 
questions to be answered are often more penetrating than mere assess- 
ment of the attainment of stated objectives* 

5* 1 Ability to formulate testable hypotheses or answerable questions in 
a research or evaluation study. 

5. 2 Eliciting evaluative questions from all important audiences for an 
evaluation, 

'This and several subsequent tasks are relevant only If an empirical, 
behavioral strategy (the most common study identified in the earlier analyses) 
is selected* ^ ^ 

13 
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' 6. Speoift^lng data or eyldenoe neaessary for a rigorous test of the hypothesis 
or an unequivooal answer to the research or evaluatloii quegtion. 

This activity* which Is important in the pursuit of any empirical researGh 

or evaluation effort, refars to general ^^s of data (e.g. , Information on 

'cost'-effectiveness of the program) rather than specifle elasses of 

variables (e, g. , oost per pupil, student gains on achievement measures) 

which are discussed elsewhere* The necessary competencies are Implicit 

in the statement of the taskp 

7. Selecting appropriate research and evaluation designs to colleot data to 
test the hypothesis or answer the questionp 

Observational techniques and mall surveys can be used to collect the same 
data, but the method used has obvious implications for the types of questions 
which c^ be answered. This activity requires not only knowledge of 
alternative designs but also a sense of their feasibility in a particular 
study* 

7. 1 Knowledge of types of designs (a, g* , experimental, quasi--experimental, 
naturalistie)* 

7. 2 Knowledge of the questions which can and cannot be answered by each . 
design. 

7.3 Knowledge of feasibility constraints (e.g. , time, access to subjects, 
control, money) which are associated with each design. 

8. Identifying the population to which results should be generallied, and 
selecting a sample of the popula tion. 

Identifying relevant populations and sampling them representatively is 
crucial to many forms of empirical research. Sociologies survey research 
is almost totally dependent for its utility upon this step having been 
successfully completed. However, in historical inqui^, historians 
seldom have control over those events which provide evidence for their 

14 



eonolusions. Educational Ustorians of the past, and to a lesser extent 
contemporary historians, are dependent upon those traces of the past 
events whieh were fortuitously left behind (In diaries, private corres- 
pondence, etc.) by a handful of eKtraordlna^ and nonrepresentatlve 
Individuals. Although historians are greatly concerned with the evidence 
which survives the rigors of time and comes into their hands, they are 
seldom forced to evaluate the generality of such e^dence in the same 
manner as the soclolo^st. - 

la evaluation, selection of a sample in such a way that ttie results 
can be generallied to a population is often of utmost importance* Although 
it is true that evaluations of particular programs often include the entire 
pppulatlon of interest, samplmg is needed when it is not feasible or 
efficient to collect data on the entire popidation. For example, it would 
scarcely be necessary to survey every parent in a large school district 
to evaluate parent reaction to the district's new busing policies* Bandom 
sampling of parents and generalising back to the population would be as 
appropriate here as in the most rigorous research study. 

8. 1 Ability to identify the population of concern. 

8.2 Ability to differentiate between theoreticEil populations and 
accessible populations. 

8. 3 ICnowledge of s^npllng theory and techniques, including variations 
on simple random lampling such as stratified sampling, cluster 
sampling, and multi-stage sampling. ^ 

Applying the research or evriuation design Md recognizing or controlling 
threats to vgdidity . 

Much empirical research involves experimental desl^s and most of the 
competencies listed below pertain directly to that type of design. This is 



probably a function of the ease with which these competerieies were identifled 
by the practlclng^Bearchers and evaluators who contributed to this 
syntheBii* Thle should not be construed as endorsement of the imbalwce 
in competencies wUch appears below but only a commentary on general 
naivete about the details to be considered in setting up other types of 
deaigns. This status is particularly troublesome because of the frequent 
cases where experimentation is inappropriate. For example, in no real 
sense is an experimental design applied in nonbehavioral research into the 
history or the philosophy of eJucation. Other designs are applied and each 
has a tQUchstone of acceptability for determining whether the results are 
reasonably ieomorphlc with reality. In ey^uatlon, the design In some 
Instances will be less sophlstlcatedj but the concern for tiireats to the 
validity of the results is Just as great. 

9, 1 Ability to recognize and eliminate or account for threats to 
validity inherent in any design under use* 

(For Experimental or Quasi-Experimental Studies, competencies 9.2 
through 9.10 are relevant) 

9*2 Knowledge of specific experimental and quasi^xperiment^ research 
designs. 

9.3 Knowledge of factors which jeopardize Internal and external validity* 

9p4 Ability to deil^ studies to control oxtraneouB variables* 

9. S Knowledge of randomization as a means of experimental control and 
its relationship to inferential statistics* 

9*6 Knowledga of fixed-effects, random-effects and mixed-effects 
designs; crossed and nested factors; the nature of interactions 
and their graphing and interpretation. 

9-7 Knowledge of the nature and problems in the use and analysis of 
"repeated measures'* designs. 
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9. 8 Knowledge of cova^ing , blocking and stratifying at means of InQreailng 
preclilon of estimation in experiments dasl^B. 

9. 9 Knowledge of the effect of measurement error on the precision {power) 
of an experiment. 

9. 10 Knowledge of pu^oiei underlying the use of randomised blocks^ Latin 
square, fractional factorial^ incomplete block deslgnSp etc* 

10# Identifying at appropriate levels of gene rallfry the goals of the program to be 
evaluatedi p ' ' : ■ -■ ■ .-^ .... . - ~ 

It cannot be assumed that the of a prop*Mi which one wishes to 

evSuate are Imown or stated in advance of the evaluation. , An Important 

activlfy in getting many evaluations under way is eliciting from the 

responsible persons the goale and objectives toward which the program 

is directed* The identification of these goals Is more than a outine 

activify of soliciting verbal statement of goals from program persoimeL 

Done properly s the identification of goals can entail some of the most 

sophisticated technolo^ of survey research and interviewing* What mmy 

e valuators e?^rlence as fmatration In their attempts to elicit statements 

of goals from program persomel Is actualfy evidence of their own lack 

of expertise in such activities* 

10* 1 Ability to Identify all audiences wMoh should help determine pro-am 
goals, 

10. 2 Ability to help others Identify their goals* 

10.3 Ability to help others prioritize their go^s. 

%criven*B (19?2) notions about goal-free evaluation were little more than a 
gleam in his rhetoric when the analyses synttoslEed herein were conducted. If 
the concept has caught on among evaluators, bm ualysls conducted today con- 
ceivably could reveal that an important task is to not identify the goals of the. 
progrMn being evaluated. 



lli Aiseseing the valua wiA teaslMlity of program goals # 

This activity is the earliest explicitly evaluative act of an evaluation. In 
an evaluation, the goali of tha program should not be accepted at face 
value, but should be regarrt^d as elements of the pro^ara appropriate 
for direcfc evaluation, much as one evaluateB program ofjerations smd 
outcomes. In some Inst^ces, the Justification of goals comes from 
empirical research in education or the soci^ scienceSp Often the 
Justification lies in a needs assessment which has been conducted 
specifically for the programp In other instances, one must turn to 
nonemplrlcal, nonbehavloral disciplines such as pbdlosophy and 
law In seeking to ev^uate program goals. Assessing the VEdue of 
program goals is an activity likely to OBLVTy the evaluator far beyond 
typical concerns with behavioral statements of objectives, criterion- 
referenced tests, and statistical analysis. It requires the evaluator 
to be educated broadly in the social sciences and philosophy and to be 
responsive to questions of value wWeh are broader than those which 
can be investigated wltMn the span of one evaluative study. 

11. 1 Ability to determine a^roprlate standards lor Jud^ng value of 
program goals. 

11.2 Ability to apply standards to program goals to determine the worth 
of those goals. 

11. 3 Ability to determine the feasibility of program goals in relation 
to resources available to try-to attain those goals. 

11. 4 Ability to determine actual and intended system outcomes to 
Identify discrepancies (needs) which exist in the system. 
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Identifying standardB qf norma for Judging worth of the phenomen.ori to be 
evaluated* 

The meaeurementB and obsei'vatlons taken in an evaluation cannot be 
translated into judgmentB of worth without itandards or nprms. The 
formality of theee staMards and norms may vary greaUy, but nonetheless 
a standard is implied whenever a Judgment of worth is derived from an 
observation. The evaluator must be sensitive to the various standards 
whlah different groups use In Judpug worth and, from among these 
standards, must either ctoose those which can best be justified or 
must use some tecluiique to eomblne and possibly weight the various 
standards* Standards may be either Internal or externali such as 
the APA Code of Ethlos (external)* or the percentile rai^ of a student 
In class (Internal). 

Translating broad objeotlves into speciflo measurable objectives* 
General goal statements often must be ope rationalized into specific 
statements of objeotlves. The onus of m^ng this translation clearly 
lies with the evaluator who possesses the teehnlaal skill for doing so and 
not on program personnel to whom the language of operatlonallzation a^d 
behaviorism is foreign and unfamiliar* Of course^ the translated objec* 
tives must be reviewed by program personnel to prevent unconscious 
biases of the evaluator from producing operational objectives different 
in intent from the broad objectives with which the program be^A. 

13. 1 Ability to state objectives in measurable terms. 

13.2 Ability to elicit and incorporate reactions of program personnel to 
statements of specific objectives* 
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14, Identi&^lng classes of variablee for measttrement * 

The Identification of behaviors for measurement is crucM In psycho- 
logical and sociologlc|Ji research on education and in educational evaluations 
wMch borrow these methods* It is far less cruci^ for historical and 
phllosophloal research. This activity is largely dependent on Imowledge 
of the phenomena being reserohed or evaluatedi The evaluator who is 
not a content specialist in the area under consideration can elicit tMs 
knowledge by combimng the same techniquas used in eliciting program 
objectives with cons Id 3 ration of the kindB of inferences called for in the 
desi^ and che standards to be used In judging the adequacy of the data 
sad Inferences, 

« 

15* Seiecting or developing techniques of measurement. 

This task Is a cornerstone of much empirical research. Coupled with 
the noKt task it constitutes the most critical stage in the pursuit of 
empirical behavioral inquiry. The evaluator also must have skill In 
selecting those techniques that will produce objective data on outcomes 
(where objective data are possible). An evaluator must know when a 
measurement technique threatens to mlsrapreeeat a set of behaviors and 
what to do to correct the problem. 

15.1 Knowledge of properties of nonaln^, ordlnalt interval, and 
measurement scales, 

15. 2 Knowledge of fimdamental theorems on the differentia weighting 
of test items, 

15.3 Knowledge of general principles of Instrument construction, 

15. 4 Knowledge of major forms of asseBsment Of knowledge s^d co^l-- 
tive skills Including multiple -choice , completionp lree*responseSs 
raking, matcMng formats, ate, 

20 
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16* 5 knowledge of prlmaty methods of asseislng attitudes and other 
affective variables, Including Llkert and Thurstone scaleB, 
semantic differentialj Q-^sort, soclometty, etc. 

15. 6 Knowledge of how to conetruct and use rating scalaB (including 

methods of assessing rater agreement), checklists, questionnaires, 
interview schedules, and observation systems. 

15* 7 AblU^ to write unambiguous items In vocabulary appropriate to 
the specified audience, 

15* 8 Ability to select appropriate standardized tests or instruments, 

15* 9 Knowledge of usee of criterion-referenced and objectives- 
referenced testing* 

16* Assessing the valldltv of measurement teotjUques . 

Scarcely any skill could be more Important to empirical Inquiry than this 

one. ta research, the validity of the findings depends in large part on skill 

in embodying the general constructs of Inquiry in a set of measurement 

techniques* As for evaluation, the worth pf an educational pro-am lies 

in Its outcomes* ® It is crucial that the proper outcomes be validly 

measured* Objective, valid data on program performance are the sine 

qua non of any justifiable evaluation* 

16* 1 Fundamental postulates and theorems of classical true^score theory. 

16* 2 Knowledge of or ability to determine instrument reliability, 
including types of reliability coefficients* 

16* 3 Knowledge of or ability to determine instrument validity, including 
various approaches to determining validity* 

16. 4 Knowledge of normlng procedures. 

16* 5 Ability to conduct item analyseii Including computing difficulty and 
discrimination indices* 



^Some would argue that the worth of a program lies in its processes or 
Its ideolo^. To each his own. 
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17. Ualn^ appropriate mettiods to Qolleot data (tests. Interviews > unobtrusive 
measurae, etc.) 

Researchers must learn well a small number of data-gathering methods 
appropriate to the problems in their discipline. Only rarely will a 
researcher's inquliy rang© so broadly as to require skill in a majority 
of the techniques for collecting data listed under earlier tasks. Not so 
the evaluator, who often is required to work across several disciplines 
and therefore' needs familiarity with a much wider variety of data 
coUeetion methods. 

17* 1 Ability to administer all data collection instruments neeessary . 
to one's research or evaluation study. 

17. 2 Ability to conduct ^1 necessa^ 'bon-instrimient" data collation 
teohniques, such as interviews and content analysis. 

18* Monitoring the program to detect deviations from design or specified 
procedures . 

It iS| of course, Important to know what one researches or evaluates. 
It is insufficient to accept mere labels when one has invested large 
portions of time and money in the observation and Judgment of outcomes. 
It is necessary that a program be monitored through site visitations, 
interview techniques, survey research methods, etc., so that the 
evaluator is clearly aware of the degree to which the proposed program 
was made operation^ or the researcher Is certain the treatment has 
been maintained. It is misleading to pronounce a Jud^ient of "unworthy" 
on a team teaching program or conclude that team teaching is inferior 
to other mathods if team teaching was never genuinely attempted. 
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19, Choosing and employing appropriate teohnlquee of statiBtlGal Malysls, 
This task subsumes an enormous number of activities. It involves 
understanding the general role, ty^s* ^d assumptions of statistioal 
techniques and drawing on such knowledge In using appropriate techniques 
for an^yElng data. Obvious^ this task is relatively ui^mportant for 
researehers in "rational" dlselpllnes, such as philosophers and 
histories* It becomes Important for empiric^ behavioral researchers 
such as sociologists and psyohologists. It is a critic^ skill for 
mathodolo^Gal researohers who are frequently e^ed upon to advise 
empiric^ researchers of many persuasions on the proper ualysls 
of research data. ^ 

The evaluator muet be broadly knowledgeable in the area of ■ 
statistical data ^alysls* A clear understanding of the pu^oses and 
general use of a varleftr of data ^atytlc techniques is crucial In most 
evaluation studies* It is important to know when a factor analysis 
bears critically on an evaluative question and when it is mere window 
dressing for a flashy.but superficial evaluation* 

19. 1 Knowledge of the general roles of statistical techniques (e. g. , 
descriptive vs. inferenti^ use of statistics). 

19. 2 Knowledge of differences in major classes of statistical 
techniques (e- g. $ Bayesl^ vs. Flsheri^ IMerencei parametric 
vs. nonparametrlc statistics) ^d princlp^ concepts associated 
with each class. 

19. 3 Knowledge of models and theories underlying statistical techniques 
(e. g* s general linear model, permutation theory, properties of 
principal probabill^ distributions). 

19. 4 Knowledge of major concepts Md use of prtoclpal statistic^ 
techniqueB (e.g., partis correlations analysis of covarianee). 
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19* 5 Knowledge of assumptions underlying-principal statlatlo^ techniques 
and consequences of lailure to meet ttese assumptions, 

19. 6 Ability to choose (or design) appropriate stEtistical techniques for 

analysis of a particular set of data, r 

19. 7 Ability to use epecjflo statistical techniques correctiy. 
20* Using electronic oomputers and computer-relnted equipment. 

The entire field of statlstloal analysis Is currentiy undergoing a slow but 
pervasive change due to the Introduction of high-speed electrorde 
computation. The researcher or evaluator who is unable to use the 
computer to advantage not only suflfers under the burden of Inefficiency 
but will soon add to it the pathos of obsolescence, 

20. 1 Ability to use computer-related equipment such as sorters, 
reproducers, or automatic test scoring machines* 

20. 2 Ability to design card layouts to aHow data analysis wltMn 
computer constralntSp and ability to use standardized computer 
programs (e. g. , BJffl series)* 

20*3 Ability to writ e com puter progrMis. 

20.4 Ability to use computer coding. 

20.5 Knowledge of capabilities of local computer systems. ^ 

20, 6 Ability to read Interpret computer output, 

21, Interpreting and drawing appropriate conclusions from data analysis. 
This task Is the heart of both evaluation and empiric^ research. Its 
importance should be apparent and needs no further comment here except 
to not^ ttat how well one can interpret data is also dependent on how 

well other steps In the study have been conducted, 

22. ^ Reporting research and evaluation finding and implications. 

This task is of primary importance for both the educational researcher 
and evaluator, ^though it Is conducted slightty differently each. 
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Science Is neQeisarlly public and henc© the act of publioatlon la Important 
in Its growth. By reporting research findings, the researcher brings 
the inquliy process full cycle* One scholar^s contribution thus enters - 
the literature to be drawn on by colleagues In the pursuit of new Imowledge. 
In a research report, detail and completeness are valued lughly wid the 
raseareher is communicating with colleagues who already have oonslderable 
e?^rlence and background in the area being discussed* The writing of the 
evaluation report is a different matter altogether. The evaluator is 
typical^ reporting to=M audience which does not share the same perspec- 
tive, grasp of teohnieal topics, nor Interest in technical details* The 
responsibility to eommunleate findings rests more heavily w3th the 
evaluator tiian with the researcher. The evaluator will have to adopt non- 
technical l^guage and must refrain from over-reliance on tabular 
presentation of data analyses. Discursive commentary on test validity 
and reliability and other topics which the evaluator' s audience will not 
find central to their concerns must be avoided. At this final stage of the 
endeavor, the evaluator will play a role much more ^In to the Journalist 
^——than-the scientist* - 

22. 1 Knowledge of the technical background and e^^erience of the 
audience (s) for the report. 

22. 2 Ability to write in a style and at a level appropriate to a specified 
audience. , 

22. 3 Ability to put qu^titatlve or numeric^ information Into verbal ^ 
or narrative form, 

22. 4 Knowledge of aiierM.te methods of presenting statistic^ data 
{e*g. , charts, graphs, or tables). 
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22.5 Knowledge of publloatlon outlets for research reporti, artioles, 
orbooks# 

22. 6 Ability to prepare and deliver an oral report of the reeearch or 
evaluation findings. ^ 

Making recommendations as a resnlt of the evgduatlon. 

The e valuator's responsibility to evaluate does not end with &e oolleotionp 

analysis^ and reporting of data* The data do not ipeak for themselves. 

Making recommendations Is as muoh a part of the evaluator's job as 

publishing is that of tiie researeher, ^ 

Providing immediate feedback on program performsnee for use in 
deoislong about program modification. 

TMs is a funetlon uiually unique to the formative evaluator Mid suj^ests 
an ongoing monitoring of the program. performance, with continuing 
feedbaok aimed at program Improvement* Although eummatlve 
evaluation and some ^^es of research can result in some form of 
feedback, seldom do these activities require the Immediacy of action 
required of the formative evaluator* 

24. 1 Ability to develop techniques for providing evaluative feedback 
to program or project personnel iii time to allow needed modlfl- 
cations to be made during the operationljf tt^ 

24. 2 Ability to identify the decision na^ere whp need eviduative feedback. 

Obt^^nigg Bnd m^a^ng regourceg fmaterial and hum^) neoeggary to 
conduct the regearch or evaluation study* 

Some regearchers and e^aluatorg do not 

"routinely=with-ftindlng=and-mana^emen^ 

actlvltleg. However, most do* It is a rare researcher or evaluator 

indeed who completes an entire career without encountering these tasks* 
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Thiy are so commonly required that it seems prudent to include the 

following competencies in the training of researchers and ev^uators* 

25* 1 Knowledge of effective techniques for writing Md submitting proposals 
to obtain funding and negotiating with funding agencies, 

25* 2 Knowledge of legalities related to research and evaluation projects. 

25.3 Abllily to determine human Md financial resources necessaiy to 
conduct a program or project and use accounting procedures to 
operate wltUn a program or project budget* 

25.4 Ability to estimate re^lstlcally the time required for research 
and evaluation activities* 

25* 5 Knowledge of and abillQr to use mana^ment and planning systems 
such as PERT OPropram Evaluatton and Bevlew Technique), PPBS 
(Program Planning Budgeting ^stem), or Critical Path Analysis. 
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III. CONCLUSION 



The lists of tasks and competencies presented in this paper are inoomplete 

will need expansion and refinement over time* Educational reeeareh and 
evaluation are rapidly advancing fields. The tasks listed herein held for the 
late sixties and early seventies. Developments subsequent to 1971 are not 
considered here, so while the tasks and competencies listed are still 
relevant* the list itself may already be dated. This seems espeel^ly 
likely in view of the recent expansions In the use of anthropologleal, 
economic, linguistic, historical, and philosophical analysis techniques 
in educational research and the introduction of new evaluation approaches 
(e. g. I Stalce, 1972) which require more attention to the many nonoognltlve 
elements of performance In evaluation work, Howevpsr, the lists presented 
herein would seem to include a majority of the tasks and competencies 
required In standard empiric^, behavioral inquiry activities which still 
seem most commonly in use in the field of education. 
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